Implement FP32 kleidiai Gemv #26302
Implement FP32 kleidiai Gemv #26302
Conversation
Signed-off-by: Jonathan Clohessy <[email protected]>
|
/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline |
|
Azure Pipelines successfully started running 4 pipeline(s). |
|
/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline |
|
Azure Pipelines successfully started running 4 pipeline(s). |
|
/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline |
|
Azure Pipelines successfully started running 4 pipeline(s). |
| kai_matmul_clamp_f32_f32p_f32p_ukernel sgemm_gemm = GetKleidiAISGemmUKernel(); | ||
| kai_matmul_clamp_f32_f32_f32p_ukernel sgemm_gemv = GetKleidiAISGemvUKernel(); |
There was a problem hiding this comment.
GetKleidiAIXUKernel() returns const&. do we need to make a copy here?
There was a problem hiding this comment.
| kai_matmul_clamp_f32_f32p_f32p_ukernel sgemm_gemm = GetKleidiAISGemmUKernel(); | |
| kai_matmul_clamp_f32_f32_f32p_ukernel sgemm_gemv = GetKleidiAISGemvUKernel(); | |
| const kai_matmul_clamp_f32_f32p_f32p_ukernel& sgemm_gemm = GetKleidiAISGemmUKernel(); | |
| const kai_matmul_clamp_f32_f32_f32p_ukernel& sgemm_gemv = GetKleidiAISGemvUKernel(); |
There was a problem hiding this comment.
updated to const in the latest push
| //No fallback and putting in guards | ||
| if(MLAS_CPUIDINFO::GetCPUIDInfo().HasArm_SME()){ | ||
| ArmKleidiAI::MlasDynamicQGemmBatch(Shape, DataParams, BatchN, ThreadPool); | ||
| if(ArmKleidiAI::SMEInfo::CanUseSME2){ |
There was a problem hiding this comment.
there are other places that need to be updated, like:
I might be missing some.
I think it would be worth making a helper function like MlasIsDynamicQGemmAvailable that has the appropriate checks and using that instead.
There was a problem hiding this comment.
Added in the updated checks in various places like these in the latest push
JonathanC-ARM
force-pushed
the
jclohess_kleidiai_gemv_implementation
branch
from
615a469 to
a3f4f5b
Compare
Signed-off-by: Jonathan Clohessy <[email protected]>
Signed-off-by: Jonathan Clohessy <[email protected]>
JonathanC-ARM
force-pushed
the
jclohess_kleidiai_gemv_implementation
branch
from
a3f4f5b to
e8ab1b1
Compare
4 participants
Description
Implementation of special sgemm path which uses GEMV kernels in cases where M or N are 1
Additionally this pr introduces the usage of a microkernel interface which utilizes typedef's provided by KleidiAI such that we can simplify the code and remove things such as ternary operations for SME1 vs SME2 kernels
Indicative Performance
In Lieu of any production models where gemv was a large contributor of the network. I opted to create a mini model to test which contains thousands of randomized matmul variants. With a distribution of GEMV cases throughout
Using onnxruntime perf test I was able to half the total inference time vs mlas with this model
More Benchmarks to come shortly